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ABSTRACT 


The purpose of the study was to evaluate the 



usefulness of teacher rating and observational techniques in the 
evaluation of a preschool program for emotionallv disturber, children. 
Eleven children (nine hoys and two girls) participated in the oroject 
occurring within the preschool program during one academic year. 
Teacher ratinas were foil to reveal that children enrolled in the 
program improved in overall functioning during the course of the 
year. Both teacher ratinas and observational techniques were dudqed 
to reflect marked gains in social development of the children, and 
were generally felt to be useful tools for evaluation of such a 
program. (Author/CD) 
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ABSTRACT 



The present project consisted of an attempt to acquire Information on the 
usefulness of teacher rating and observational techniques in the evaluation of 
a preschool program for emotionally disturbed children. Teacher ratings revealed 
that children enrolled In this program Improved In overall functioning during 
the course of one academic year. Both teacher ratings and observational 
techniques reflected marked gains In social development. 
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EVALUATION OF A SPECIALIZED NURSERY SCHOOL PROGRAM 
FOR EMOTIONALLY DISTURBED CHILDREN* 

Linda Bllsky 

Teachers College, Columbia University 

Recently, the problem of evaluating preschool programs has become an Issue 
with far-reaching political, as well as educational, Implications. Not only have 
the results of evaluations been expected to provide guidelines for Improving 
educational practices within existing programs, but also these results have been 
expected to provide justification for the existence of the programs themselves. 
The situation has been complicated by the absence of appropriate evaluation 
techniques of established effectiveness. In the case of the present project, 
which consisted uf the evaluation of a fairly unique type of preschcdl program, 
existing techniques were strikingly Inadequate, Consequently, It was necessary 
to develop evaluation techniques which were specifically appropriate for this 
type of preschool program. 

The program constituted the central thrust of a multidisciplinary effort 
to Improve the functioning of preschool children who were experiencing serious 
emotional and social difficulties, A variety of techniques were employed with 
small groups of children In an attempt to provide a therapeutic nursery school 
program, The need to evaluate the effectiveness of tills program led to the 
present project which endeavored to develop techniques which (a) would detect 
behavior changes occurring within this program during one academic year and 
(b) would Le appropriate for the evaluation of similar programs, 

Previous evaluations of preschool programs have generally described pro- 
gress In terms of measures of cognitive development, such as IQ, language 

*The work presented or reported herein was performed pursuant to a 
grant from the U.S, Office of Education, Department of Health, Education 
and Welfare, 
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development, or reading readiness (01 Lorenzo & Salter, 1968; Gray & Klaus, 

1965), However, the nature of the present program demanded the measurement 
of a much broader range of performance, which would Include social and emo- 
tional as well as cognitive behavior. Teacher rating techniques have often 
been employed as measures of social and emotional adjustment In the classroom, 
However, although teacher observations are an extremely Important source of 
Information about the child's day to day behavior, they are Inevitably influ- 
enced by the teacher's relationship with each child and the teacher's Involvement 
with the goals of the program. Thus, the present project sought to supplement 
teacher ratings with an additional, more objective, measure of behavior. 

Wright (1967) outlined a method which employed trained observers to record 
fairly complete descriptions of an Individual child's behavior during a 
specified period of time. Although Wright's method was Intended to provide 
ecological data on child behavior, similar techniques have been adapted to 
provide relatively objective Information for use In the evaluation of class- 
room programs, Werry and Quay (1969) successfully employed an observational 
technique In the evaluation of an elementary school program for emotionally 
disturbed children, Werry and Quay's technique consisted of recording the 
frequency of behaviors In certain pre-selected categories during a specified 
period of time. Raph, Thomas, Chess, and Korn (1968) trained observers to 
record the behavior of children In a nursery school program for specified 
periods of time, They subsequently analyzed these narrative accounts In terms 
of types of social Interactions, The present project sought to employ a 
corrblnation of rating scale and observational jechnlques which would provide an 
objective, as well as comprehensive, record of behavior occurring within a 
preschool program, 
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METHOD 

Subjects. Eleven children (9 boys and 2 girls) participated in the 
present evaluation project. All children were experiencing severe emotional 
and social difficulties associated with manifestations of brain damage, mental 
retardation and/or other complicating factors. The group ranged In chrono- 
logical age from 47 to 73 months with a mean of 57.18 months, four of the 
children were participating In their first year of the nursery school program 
and the remaining seven were participating in their second year. Psychological 
evaluations were administered to all children during the first month of the 
academic year. Four children were judged to be untestable. The remaining 
children obtained IQs ranging from dull normal to superior on the Merrill 
Palmer Scale. 

The Program . The present evaluation project was chiefly concerned with 
performance within a nursery school program which was designed as the central 
part of a multidisciplinary treatment effort. The nursery school program 
consisted of two classes, each conducted by one teacher and one assistant 
teacher, which .net for five hours four days a week and for two and a half hours 
on the fifth day. Three children were enrolled In one class and six children 
were enrolled in a second class throughout the year; one child was transferred 
from the first class to the second class after five months; another child was 
transferred from the first class to a normal nursery school class after six 
months. 

Both classes followed a relatively structured nursery school routine which 
Included activities, described by Fouracre, Connor, and Goldberg (1962); sw.it! of 
these activities were arts and crafts, music, listening to stories, free play, 
gym, outdoor play, swinging, rest periods, and lunch. Gym and outdoor play 
periods were conducted Jointly with several normal nursery school classes, This 
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curriculum was utilized to provide opportunities for therapeutic, emotional and 
social experiences. Activities were structured so that the children could (a) 
learn to relate to others through shared experiences and (b) increase their 
personal competence by acquiring a variety of new skills. Itoreover, the nursery 
school activities provided a setting for the implementation of the chief program 
goal: to improve social and emotional functioning. 

The two classes were part of an overall treatment effort designed and 
directed by a psychiatrist, Additional services were provided by a multidisci- 
plinary staff which met as a group at least twice weekly. This staff included a 
psychologist who provided evaluational services, a language therapist who 
provided regular language training for most of the children, and a psychiatric 
resident who provided individual therapy for one of the children. Several social 
workers provided counseling for all parents at least once a week and, In addition, 
conducted a mothers' group which net weekly. 

Evaluation Techniques . The evaluational strategy for the present project 
consisted of an attempt to describe behavioral changes occurring within the pre- 
school program during one academic year, Descriptions of each child's behavior 
were obtained from two sources, (a) rating scales completed by the child's 
teacher and (b) observation reports provided by a team of outside observers. 

The rating scale techniques Involves the use of two rating scales, These 
scales, designed to measure a child's typical classroom behavior as observed by 
his teacher, were (a) the Behavior Rating Scale and (b) the Curriculum Guide 
Rating Scale, The behavior Rating Scale, which emphasized social and emotional 
behavior, was developed for the present project In order to supplement the Curri- 
culum Guide Rating Scale, which provided a more general description of behavior. 
The Behavior Rating Scale contained fourteen Items reflecting selected behav- 
ioral objectives of the present nursery school program. Each Item contained 
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five statements descriptive of typical behavior related to these objectives. 
These statements were rank ordered from one to five In terms of Increasing 
social desirability. The rater Indicated which cf the five statements best 
described the child's typical behavior In each Instance, The Curriculum Guide 
Rating Scale was modified for the present project from that described by Four- 
acre, Connor, and Goldberg (1962), After modifications, it consisted of six 
sections classified as Intellectual Development (26 Items), Imaginative and 
Creative Expression (12 items), Social Development (16 Items), Manipulative 
Development (17 Items), Motor Development (12 Items), and Self Help (27 Items), 
Each Item was rated on a scale from zero to five, with five representing 
optimal performance on an Item, Both the behavior Rating Scale and the Curri- 
culum Guide Rating Scale were completed for each child In the Fall and again In 
the Spring, The Spring ratings v/ere completed without reference to the Fall 
ratings. 

Information based on teacher ratings was supplemented by Information based 
on a series of observations which took place In the Fall and Spring of the same 
academic year. All observers were graduate students enrolled In Psychology or 
Education programs. The 22 Fall observers participated as a course requirement, 
whereas the 14 Spring observers participated on a volunteer basis, Ten minute 
observation sessions v/ere scheduled, according to the availability of observers, 
so that tney occurred within approximately seven weeks In the Fall and In the 
Spring. An attempt was made to schedule observations during as wide a variety 
of dally activities as possible. Observers v/ere assigned to children randomly 
without replacement, with the restriction that an observer should observe the 
SAme child no more than twice. Observation reports with missing Information 
were discarded, leaving a pool of 14B Fall reports and 83 Spring reports from 
which six Fall reports and six Spring reports for each child were randomly 
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selected. These constituted the 132 reports used in the analyses. 

All observers wert instructed to unselectlvely record as much as possible 
of an individual child's behavior for ten minutes. Interpretations uere to be 
avoided, Observers were required to submit an initial practice observation 
report which was screened for procedural deviations before the remaining observa- 
tions took place. Observations of classroom behavior were conducted from 
observation booths with one-way vision screens and sound amplifying equipment, 
When observations took place outside the classroom, observers were instructed to 
remain as unobtrusive as possible. 

All reports selected for analysis were summarized (by non-participant 
raters) according to a classification system developed for the present project. 
First, each report was divided into units of behavior, defined as behavior 
segments involving a change of subject, object, or type of behavior. Units were 
then classified as representing one of the following types of behavior: 

1, E-A Emotional behavior with apparent cause 

2, E-IIA Emotional behavior with no apparent cause 

3, 1-V Social interaction involving verbal contact only 

4, I-C Social interaction involving physical contact only 

5, 1-VC Social interaction Involving both verbal and pliysical 

contact 

6, A-P Activity which appears purposeful or goal -directed 

7, A-fiP Activity which does not appear purposeful or goal-directed 

Behaviors in each of the above categories were then rated as either self- 

initiated or other-initiated*, either positive, negative, or neutral In emotional 
tonej and occurring either with self only, a peer only, an adult only, or a 
group (i.e,, more than one peer and/or adult). 

The classification system was applied by two raters i the first rated 22 
nndomly selected reports vihlch consisted of one Fall report and one Spring 
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report per child; and the second rated the 110 remaining reports which consisted 
of five Fall reports and five Spring reports per child. Both raters Initially 
rated several practice reports independently and then negotiated any differences. 
In addition, each rater Independently rated ten randomly selected reports which 
had been rated by the other rater so that reliability could be established. 

Dates and names of children had been removed from reports so that this Infor- 
mation would not Influence ratings. 



RESULTS 

Teacher Rating Scales. Sign Tests (two-tailed) were applied to each of the 
fourteen Behavior Rating Scale Items In order to determine the significance of 
Fall-Spring changes as reported by the teachers. Positive changes, significant 
at the .05 level, were obtained on two Items: Interaction with others, and 
participation In group activities. 

Sign Tests were also applied to each section of the Curriculum Guide Rating 
Scale to test for Fall-Spring differences as seen by the teachers. Significant 
differences were revealed for all six sections of the Curriculum Guide Rating 
Scale. Improvement, significant at the ,01 level, was observed In the areas of 
imaginative and creative expression, social development, and motor development. 
Improvement, significant at the .05 level, was observed in the areas of intel- 
lectual development, manipulative development, and self help. 

Observations. Inter-rater reliabilities were based on 20 observation 
reports which had been rated Independently by both raters. Spearman Rank Order 
correlation coefficients between raters were obtained for each major behavior 
category. The Inter-rater correlation for the category of negative behavior was 
significant at the .05 level ( rho g .53). Correlations for all other categories 
were significant at the .01 level: emotional behavior ( rho ■ .85), social Inter- 
action (rho a .89), activity (rho ■ .91), self-initiated behavior (rho « .96), 
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other-initiated behavior (rho ■ .80), positive behavior ( rho = ,85), neutral 

behavior (rho 3 .90), behavior with self (rho B .96), behavior with peer 

(rho ■ .87), behavior with adult ( rho ° .93), and behavior with group (rho = .92). 

It was Initially determined that the number of behavior units rated in the 
Spring did not differ from the number of behavior units rated In tho Fall 
(t. °.55, df = 10, £ .05). Subsequently, Sign Tests were performed to test 

for Fall to Spring changes In the frequency of behaviors classified In each 
ma.lor category. An Increase, from Fall to Spring, In behaviors classified as 
social Interactions was significant at the .05 level. None of the other 
categories revealed significant changes In frequency from Fall to Spring. 

DISCUSSION 

The results of the present project indicated that a number of positive 
behavior changes had occurred within the specialized nursery school program for 
emotionally disturbed children. The most striking changes, as seen by both the 
teachers and the outside observers, occurred within the area of social develop- 
ment. These findings were especially encouraging, because an effort to improve 
social functioning had constituted a central emphasis of the nursery school 
program. Additional results, based on the Curriculum Guide Hating Scale, 
completed by the teachers, Indicated general improvement in several other areas 
including Intellectual development, Imaginative and creative expression, mani- 
pulative development, motor development, and self help skills. 

Overall, these findings suggested that the children within the nursery 
school program were making significant progress In a number of Important areas. 
However, because of the unavailability of a suitable control group, the possi- 
bility cannot be ruled out that factors other than the teaching approaches 
employed in the nursery school program may have contributed to the behavior 
O 
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changes occurring within the program. On the other hand, the information 
obtained should be useful in (a) describing behavior which typically occurs 
within this type of program, (b) indicating behaviors which tended to Increase 
in frequency and indicating behaviors which tended to remain fairly stable 
during an academic year, (c) provld' g guidelines for the establishment of 
teaching priorities within the present preschool program as well as similar 
programs, and (d) providing information on potentially useful evaluation methods 
for future use in the present program and similar programs. 
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